System and method for electronic data communication

ABSTRACT

In a method for transmitting video for a display panel between a transmitter in electronic communication with a receiver over a wireless communication channel, the method includes: receiving, by a transmitter, a data signal from a data source; receiving, by the transmitter, a return signal from a receiver; selecting, by the transmitter based on at least one of channel quality, video quality, codec requirements, or data rate requirements, a profile from among a plurality of profiles each comprising one or more parameters for transmitting the data signal to the receiver, the plurality of profiles comprising one or more profiles corresponding to transmission of uncompressed video data and one or more profiles corresponding to transmission of compressed video data; and transmitting, by the transmitter, the data signal to the receiver according to the selected profile for display on the display panel.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/359,553, filed Nov. 22, 2016 which claims priority to and the benefitof U.S. Provisional Patent Application No. 62/400,042, entitled “ASYSTEM AND METHOD FOR CROSS LAYER IMAGE OPTIMIZATION (CLIO) FOR WIRELESSVIDEO TRANSMISSION OVER MULTI-GIGABIT CHANNELS”, filed in the UnitedStates Patent and Trademark Office on Sep. 26, 2016, the entire contentsof both of which are incorporated herein by reference. The presentapplication is further related to U.S. patent application Ser. No.15/359,551, filed Nov. 22, 2016 entitled “System and Method forElectronic Data Communication”, and U.S. patent application Ser. No.15/359,564, filed Nov. 22, 2016 entitled “System and Method forElectronic Data Communication”, both filed on even date herewith, theentire contents of both of which are incorporated herein by reference.

FIELD

Aspects of one or more example embodiments of the present inventionrelate to a system and method for electronic data communication.

BACKGROUND

The demand for video transmission over wireless is increasing due toadvent of new applications and use cases. Technological advancements inhigh resolution display screens and the advent of high quality video(HD, FHD, UHD, etc.) have resulted in increased bandwidth requirementsfor high throughput transmissions over recent years. For example,uncompressed Ultra High Definition (UHD) video requires 12 Gbps ofbandwidth.

In addition to high data rate constraints, wireless video transmissionis also time/delay sensitive. When provided with 60 frames per second,the inter-frame time is 1/60=16.6 ms. Thus, any portion of a frame notreceived within 16.6 ms must be dropped so that the display can beginrendering the next frame, and data retransmission is typically not aviable option. In addition to high bandwidth and latency requirements,wireless channels may be susceptible to interference, which may causethe quality of a wireless channel to vary unpredictably over time. Thus,it may be difficult to provide a guaranteed Quality of Service (QoS) forsending high quality video data over wireless channel.

The above information discussed in this Background section is only forenhancement of understanding of the background of the describedtechnology and therefore it may contain information that does notconstitute prior art that is already known to a person having ordinaryskill in the art.

SUMMARY

Aspects of one or more example embodiments of the present inventionrelate to a system and method for electronic data communication.

According to some example embodiments, in a method for transmittingvideo for a display panel between a transmitter in electroniccommunication with a receiver over a wireless communication channel, themethod includes: receiving, by a transmitter, a data signal from a datasource; receiving, by the transmitter, a return signal from a receiver;selecting, by the transmitter based on at least one of channel quality,video quality, codec requirements, or data rate requirements, a profilefrom among a plurality of profiles each comprising one or moreparameters for transmitting the data signal to the receiver, theplurality of profiles comprising one or more profiles corresponding totransmission of uncompressed video data and one or more profilescorresponding to transmission of compressed video data; andtransmitting, by the transmitter, the data signal to the receiveraccording to the selected profile for display on the display panel.

According to some embodiments, the data signal received by thetransmitter is a video data signal.

According to some embodiments, the method further includes selecting theprofile from among the plurality of profiles based on the return signal,wherein the return signal comprises an indicator of a quality of thewireless communication channel as measured by the receiver.

According to some embodiments, the method further includes selecting theprofile from among the plurality of profiles based on the return signal,wherein the return signal comprises an indicator of visual quality asmeasured by the receiver.

According to some embodiments, the method further includes identifying,based on the return signal, a display device corresponding to thereceiver; and selecting, by the transmitter, the profile based on thedisplay device corresponding to the receiver.

According to some embodiments, the plurality of profiles comprises oneor more of the following: a first profile defining parameters fortransmitting uncompressed video data in which a channel quality exceedsa first threshold level, such that if pixel data from any pixel from agroup of pixels is lost at the receiver after transmission, the receiverrecalculates the lost pixel data by averaging values of surroundingpixels; a second profile defining parameters for transmittinguncompressed video data in which a channel quality is below the firstthreshold level but above a second threshold level, such that the pixeldata from one pixel of a group of pixels is not transmitted to thereceiver and the receiver recalculates the pixel data from the one pixelby averaging values of surrounding pixels; a third profile definingparameters for transmitting uncompressed video data in which a channelquality is below the first threshold level and the second thresholdlevel but above a third threshold level, such that a least significantbit of pixels in a packet is not transmitted to the receiver and thereceiver compensates for the least significant bit; fourth, fifth,sixth, and seventh profiles defining parameters for transmittingcompressed video data compressed using a layer-based compression methodthat generates a plurality of data layers, the plurality of data layerscomprising a highest layer having a first compression ratio, and alowest layer having data that, if added to the highest layer and anyintervening layers has a second compression ratio lower than the firstcompression ratio, wherein: according to the fourth profile, each of thedata layers is transmitted to the receiver; according to the fifthprofile, a subset of the data layers is transmitted to the receiver;according to the sixth profile, a single data layer from among the datalayers is transmitted to the receiver; and according to the seventhprofile, only a portion of one or more of the data layers is transmittedto the receiver; and an eighth profile defining parameters fortransmitting video data with different priority levels based on adisplay device for displaying the video data.

According to some embodiments, the method further includes transmitting,by the transmitter, information regarding the selected profile to thereceiver by a bit selection in a header packet of the data signal.

According to some embodiments, the information regarding the selectedprofile comprises at least one of decompression, reverse layering,decoding, or error correction information corresponding to the selectedprofile.

According to some example embodiments of the present invention, in atransmitter for transmitting data for a display panel to a receiver overa wireless communication channel, the transmitter is configured to:receive a data signal from a data source; receive a return signal from areceiver; select, based on at least one of channel quality, videoquality, codec requirements, or data rate requirements, a profile fromamong a plurality of profiles each comprising one or more parameters fortransmitting the data signal to the receiver, the plurality of profilescomprising one or more profiles corresponding to transmission ofuncompressed video data and one or more profiles corresponding totransmission of compressed video data; and transmit the data signal tothe receiver according to the selected profile for display on thedisplay panel.

According to some embodiments, the data signal received by thetransmitter is a video data signal.

According to some embodiments, the transmitter is further configured toselect the profile from among the plurality of profiles based on thereturn signal, wherein the return signal comprises an indicator of aquality of the wireless communication channel as measured by thereceiver.

According to some embodiments, the transmitter is further configured to:identify, based on the return signal, a display device corresponding tothe receiver; and select the profile based on the display devicecorresponding to the receiver.

According to some embodiments, the plurality of profiles includes one ormore of the following: a first profile defining parameters fortransmitting uncompressed video data in which a channel quality exceedsa first threshold level, such that if pixel data from any pixel from agroup of pixels is lost at the receiver after transmission, the receiverrecalculates the lost pixel data by averaging values of surroundingpixels; a second profile defining parameters for transmittinguncompressed video data in which a channel quality is below the firstthreshold level but above a second threshold level, such that the pixeldata from one pixel of a group of pixels is not transmitted to thereceiver and the receiver recalculates the pixel data from the one pixelby averaging values of surrounding pixels; a third profile definingparameters for transmitting uncompressed video data in which a channelquality is below the first threshold level and the second thresholdlevel but above a third threshold level, such that a least significantbit of pixels in a packet is not transmitted to the receiver and thereceiver compensates for the least significant bit; fourth, fifth,sixth, and seventh profiles defining parameters for transmittingcompressed video data compressed using a layer-based compression methodthat generates a plurality of data layers, the plurality of data layerscomprising a highest layer having a first compression ratio, and alowest layer having data that, if added to the highest layer and anyintervening layers has a second compression ratio lower than the firstcompression ratio, wherein: according to the fourth profile, each of thedata layers is transmitted to the receiver; according to the fifthprofile, a subset of the data layers is transmitted to the receiver;according to the sixth profile, a single data layer from among the datalayers is transmitted to the receiver; and according to the seventhprofile, only a portion of one or more of the data layers is transmittedto the receiver; and an eighth profile defining parameters fortransmitting video data with different priority levels based on adisplay device for displaying the video data.

According to some embodiments, the transmitter is further configured totransmit information regarding the selected profile to the receiver by abit selection in a header packet of the data signal, wherein theinformation regarding the selected profile comprises at least one ofdecompression, reverse layering, decoding, or error correctioninformation corresponding to the selected profile.

According to some embodiments of the present invention, in a method fortransmitting video for a display panel between a transmitter inelectronic communication with a receiver over a wireless communicationchannel, the method includes: receiving, by a transmitter, a data signalfrom a data source; receiving, by the transmitter, a return signal froma receiver; selecting, by the transmitter, a profile from among aplurality of profiles each comprising one or more parameters fortransmitting the data signal to the receiver, the plurality of profilescomprising one or more profiles corresponding to transmission ofuncompressed video data and one or more profiles corresponding totransmission of compressed video data; and transmitting, by thetransmitter, the data signal to the receiver according to the selectedprofile for display on the display panel.

According to some embodiments, the method further includes selecting theprofile based on channel quality.

According to some embodiments, the method further includes selecting theprofile based on video quality as measured at the receiver.

According to some embodiments, the method further includes selecting theprofile based on codec requirements.

According to some embodiments, the method further includes selecting theprofile based on data rate requirements.

According to some embodiments, the method further includes identifying,based on the return signal, a display device corresponding to thereceiver; and selecting, by the transmitter, the profile based on thedisplay device corresponding to the receiver.

According to some embodiments, the method further includes monitoring,by the transmitter, the return signal from the receiver; and selecting,by the transmitter, another profile for subsequent transmission based ona change in at least one of channel quality or video quality.

According to some example embodiments of the present invention, inmethod for transmitting video for a display panel between a transmitterin electronic communication with a receiver over a wirelesscommunication channel, the method includes: receiving, by a transmitter,a frame of video data from a data source; grouping, by the transmitter,bits corresponding to the frame of video data into a plurality of groupseach corresponding to a plurality of levels of importance; reorganizing,by the transmitter, the groups in order of importance to generate areorganized frame of data with a group corresponding to a plurality ofpacket headers having a highest level of importance and arranged to befirst among the groups; inserting, by the transmitter, a valueindicating a length of data bits corresponding to each packet headerbefore each packet header; and transmitting, by the transmitter, thebits corresponding to the frame of video data to the receiver fordisplay on the display panel such that each group from among theplurality of groups is transmitted according to different protectiontechniques based on their corresponding levels of importance.

According to some embodiments, each group has a different modulation andcoding scheme (MCS) value based on their corresponding level ofimportance.

According to some embodiments, each group has a different forward errorcorrection coding rate based on their corresponding level of importance.

According to some embodiments, a decoder at the receiver is configuredto reconstruct the frame of video data.

According to some embodiments, reconstructing the frame of video datacomprises moving each packet header to their original relativelocations.

According to some embodiments, reconstructing the frame of video datacomprises removing the value indicating the length of data bits.

According to some embodiments, the method further includes monitoring,by the transmitter, a return signal from the receiver; and adjusting, bythe transmitter, the different protection techniques for subsequenttransmission.

According to some embodiments, the method further includes selecting, bythe transmitter, the different protection techniques according to a typeof display device corresponding to the receiver.

According to some example embodiments of the present invention, in atransmitter for transmitting data for a display panel to a receiver overa wireless communication channel, the transmitter is configured to:receive a frame of video data from a data source; group bitscorresponding to the frame of video data into a plurality of groups eachcorresponding to a plurality of levels of importance; reorganize thegroups in order of importance to generate a reorganized frame of datawith a group corresponding to a plurality of packet headers having ahighest level of importance and arranged to be first among the groups;insert a value indicating a length of data bits corresponding to eachpacket header before each packet header; and transmit the bitscorresponding to the frame of video data to the receiver for display onthe display panel such that each group from among the plurality ofgroups is transmitted according to different protection techniques basedon their corresponding levels of importance.

According to some embodiments, each group has a different modulation andcoding scheme (MCS) value based on their corresponding level ofimportance.

According to some embodiments, each group has a different forward errorcorrection coding rate based on their corresponding level of importance.

According to some embodiments, a decoder at the receiver is configuredto reconstruct the frame of video data by moving each packet header totheir original relative locations.

According to some embodiments, the transmitter is further configured to:monitor a return signal from the receiver; and adjust the differentprotection techniques for subsequent transmission.

According to some example embodiments of the present invention, in amethod for transmitting video for a display panel between a transmitterin electronic communication with a receiver over a wirelesscommunication channel, the method includes: receiving, by a transmitter,a frame of video data from a data source; grouping, by the transmitter,bits corresponding to the frame of video data into a plurality of groupseach corresponding to a plurality of levels of importance; reorganizing,by the transmitter, the groups in order of importance to generate areorganized frame of data with a group corresponding to a plurality ofpacket headers having a highest level of importance and arranged to befirst among the groups; and transmitting, by the transmitter, the bitscorresponding to the frame of video data to the receiver for display onthe display panel such that each group from among the plurality ofgroups is transmitted according to different protection techniques basedon their corresponding levels of importance.

According to some embodiments, the method further includes inserting, bythe transmitter, a value indicating a length of data bits correspondingto each packet header before each packet header.

According to some embodiments, each group has a different modulation andcoding scheme (MCS) value based on their corresponding level ofimportance.

According to some embodiments, each group has a different forward errorcorrection coding rate based on their corresponding level of importance.

According to some embodiments, a decoder at the receiver is configuredto reconstruct the frame of video data.

According to some embodiments, the method further includes: monitoring,by the transmitter, a return signal from the receiver; and adjusting, bythe transmitter, the different protection techniques based on a changein at least one of channel quality or video quality for subsequenttransmission.

According to some embodiments, the method further includes selecting, bythe transmitter, the different protection techniques for datacorresponding to different regions of an image according to a type ofdisplay device corresponding to the receiver.

According to some example embodiments of the present invention, in amethod for transmitting video for a display panel between a transmitterin electronic communication with a receiver over a wirelesscommunication channel, the method includes: receiving, by a transmitter,a frame of video data from a data source; reorganizing, by thetransmitter, the frame of video data into a plurality of packetsaccording to levels of importance of bits of the frame of video data;generating, by the transmitter, a tag for each of the packets, the tagscorresponding to different relatively levels of importance of thepackets; performing, by the transmitter, different protection techniquesfor each of the packets based on the tag corresponding to each of thepackets; and transmitting, by the transmitter, the packets and the tagsto the receiver for display on the display panel such that each packetis transmitted according to the different protection techniques based ontheir corresponding tagging.

According to some embodiments, the tag of each of the packets comprisesadding a bit pattern indicating a relative importance level of a packetto a header of the packet.

According to some embodiments, the tag of each packet corresponds to amodulation and coding scheme (MCS) value based on their correspondinglevel of importance.

According to some embodiments, more important packets have a lower MCSvalue than less important packets.

According to some embodiments, the tag of each packet corresponds to aforward error correction coding rate based on their corresponding levelof importance.

According to some embodiments, the method further includes: packing, bythe transmitter, the packets into a plurality of Aggregated MAC ProtocolData Unit (A-MPDU) subframes; and transmitting, by the transmitter, theA-MPDU subframes to the receiver.

According to some embodiments, the method further includes tagging, bythe transmitter, a plurality of Physical Layer Convergence Procedure(PLCP) Protocol Data Unit (PPDU) frames with a same tag; packing, by thetransmitter, the A-MPDU subframes having a same importance level intothe PPDU frames; and transmitting, by the transmitter, the PPDU framesto the receiver.

According to some embodiments, the method further includes selecting, bythe transmitter, the different protection techniques for differentregions of an image of the frame of video data according to a type ofdisplay device corresponding to the receiver.

According to some embodiments, in a transmitter for transmitting datafor a display panel to a receiver over a wireless communication channel,the transmitter is configured to: receive a frame of video data from adata source; reorganize the frame of video data into a plurality ofpackets according to levels of importance of bits of the frame of videodata; generate a tag for each of the packets, the tags corresponding todifferent relatively levels of importance of the packets; performdifferent protection techniques for each of the packets based on the tagcorresponding to each of the packets; and transmit the packets and thetags to the receiver for display on the display panel such that eachpacket is transmitted according to the different protection techniquesbased on their corresponding tagging.

According to some embodiments, the tagging of each of the packetscomprises adding a bit pattern indicating a relative importance level ofa packet to a header of the packet.

According to some embodiments, the tagging of each packet corresponds toa modulation and coding scheme (MCS) value based on their correspondinglevel of importance.

According to some embodiments, the tagging of each packet corresponds toa forward error correction coding rate based on their correspondinglevel of importance.

According to some embodiments, the transmitter is further configured to:pack the packets into a plurality of Aggregated MAC Protocol Data Unit(A-MPDU) subframes; and transmit the A-MPDU subframes to the receiver.

According to some embodiments, the transmitter is further configured to:tag a plurality of Physical Layer Convergence Procedure (PLCP) ProtocolData Unit (PPDU) frames; pack the A-MPDU subframes having a sameimportance level into the PPDU frames; and transmit the PPDU frames tothe receiver.

According to some example embodiments of the present invention, in amethod for transmitting video for a display panel between a transmitterin electronic communication with a receiver over a wirelesscommunication channel, the method includes: receiving, by a transmitter,a frame of video data from a data source; reorganizing, by thetransmitter, the frame of video data into a plurality of packetsaccording to levels of importance of bits of the frame of video data;generating, by the transmitter, a tag for each of the packets, the tagscorresponding to different relatively levels of importance of thepackets; and transmitting, by the transmitter, the packets and the tagsto the receiver for display on the display panel such that each packetis transmitted according to different protection techniques based ontheir corresponding tagging.

According to some embodiments, the method further includes performing,by the transmitter, different protection techniques for each of thepackets based on the tag corresponding to each of the packets.

According to some embodiments, the tag of each of the packets includesadding a bit pattern indicating a relative importance level of a packetto a header of the packet.

According to some embodiments, the method further includes packing, bythe transmitter, the packets into a plurality of Aggregated MAC ProtocolData Unit (A-MPDU) subframes; and transmitting, by the transmitter, theA-MPDU subframes to the receiver.

According to some embodiments, the method further includes tagging, bythe transmitter, a plurality of Physical Layer Convergence Procedure(PLCP) Protocol Data Unit (PPDU) frames with a same tag; packing, by thetransmitter, the A-MPDU subframes having a same importance level intothe PPDU frames; and transmitting, by the transmitter, the PPDU framesto the receiver.

According to some embodiments, the method further includes monitoring,by the transmitter, a return signal from the receiver; and adjusting, bythe transmitter, the different protection techniques based on at leaston a change in at least one of channel quality or video quality.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

A more complete appreciation of the present invention, and many of theattendant features and aspects thereof, will become more readilyapparent as the invention becomes better understood by reference to thefollowing detailed description when considered in conjunction with theaccompanying drawings in which like reference symbols indicate likecomponents, wherein:

FIG. 1 is a block diagram illustrating a wireless data transmissionsystem illustrating a high level overview description of a cross-layeroptimization system, according to some example embodiments of thepresent invention;

FIG. 2 illustrates an example layer-based compression scheme utilizedaccording to some example embodiments of the present invention;

FIG. 3 is a block diagram illustrating further details of an examplearchitecture and some components of a cross-layer optimization system,according to some example embodiments of the present invention;

FIG. 4 illustrates an example partitioning of pixels into differenttypes, according to some example embodiments of the present invention;

FIG. 5 illustrates an example of a field of view for one eye, accordingto some example embodiments of the present invention;

FIG. 6 illustrates an example of a reorganization of a packet or layerstructure, according to some example embodiments of the presentinvention;

FIG. 7 illustrates an example header structure according to some exampleembodiments of the present invention;

FIG. 8 illustrates an example reorganization of data, according to someexample embodiments of the present invention;

FIG. 9 illustrates an example reorganization of pixel bits and packets,according to some example embodiments of the present invention;

FIG. 10 illustrates available header fields under a wirelesscommunication standard; and

FIG. 11 is a flow chart illustrating a process for cross-layer imageoptimization, according to some example embodiments of the presentinvention.

DETAILED DESCRIPTION

Aspects of example embodiments of the present invention relate to asystem and method for electronic data communication.

The detailed description set forth below in connection with the appendeddrawing is intended as a description of exemplary embodiments of asystem and method for cross layer image optimization (CLIO) for wirelessvideo transmission over multi-gigabit channels, provided in accordancewith the present invention and is not intended to represent the onlyforms in which the present invention may be constructed or utilized. Thedescription sets forth the features of the present invention inconnection with the illustrated embodiments. It is to be understood,however, that the same or equivalent functions and structures may beaccomplished by different embodiments that are also intended to beencompassed within the spirit and scope of the invention. As denotedelsewhere herein, like element numbers are intended to indicate likeelements or features.

The future of display technology includes a world full of inexpensivedisplays fed by various wireless streaming devices (mobile phones,set-top boxes, projectors, etc.). High quality video transmission overwireless links has shown to be challenging. Wireless devices arenon-stationary, while wireless links have scarce bandwidth and aresusceptible to many types of noise. The latency may also be high andvariable, which is particularly harmful to video. Due to the stringentrequirements for video transmission, the common design methodology inwhich different layers (e.g., the Application (APP) layer, media accesscontrol (MAC) layer, and Physical (PHY) layer) are designedindependently does not facilitate high data rate wireless datatransmission. Accordingly, embodiments of the present invention providea cross layer approach, in which the information at one layer isutilized to change parameters at different layers. Such flexibilityallows for quick adaptation to the fast changes of the wireless link.

The IEEE 802.11ad standard is capable of providing the required bitratefor uncompressed full high definition (FHD) wireless video. 802.11adworks in the 60 GHz band using channels with 2.16 GHz of bandwidth andprovides up to 4.6 Gbps of bandwidth at the physical layer (PHY) using asingle carrier, which is sufficient for uncompressed FHD videotransmission. IEEE 802.11ad, however, is only able to obtain the maximumbandwidth in certain deployments. For example, IEEE 802.11ad requiresthe transmitter and receiver to be located within short distance of eachother and within line of sight (LoS)). Embodiments of the presentinvention, therefore, provide an improved approach for wireless datatransmission.

According to several embodiments, various characteristics of the presentinvention may be used to improve and guarantee the QoS for videostreaming over wireless networks, including Physical (PHY), mediumaccess control (MAC), or Application (APP) cross layer solutions.Accordingly, information from one layer (for example, the MAC layer) maybe used to optimize parameters in another layer (for example, the APPlayer). For example, in video streaming, the APP layer could utilizeinformation about channel quality in rate control (network awareness).Lower layers may also be configured to utilize information about videotraffic characteristics. According to various embodiments, the systememploys dynamic profile partitioning, dynamic tagging for packets,unequal error protection for different layers in layer based compressiontechnique, and importance level aware modulation and modulation/codingselection, packetization, and bit or pixel pruning.

According to one embodiment, a cross layer method may be utilized tooptimize perceptual quality in delay constrained scalable videotransmission. Additionally, Unequal Error Protection (UEP) may beemployed according to packet loss visibility in the PHY layer for eachvideo layer. There is also buffer-aware source adaptation in the APPlayer. A rate adaptation scheme is used for QoS-driven seamless handoffscheme based on the IEEE 802.11 Media Independent Handover (MIH)framework. In order to control the rate, the Quantization Parameter (QP)is adapted for the single layer coding (AVC/H.264) and the enhancementlayers are dropped for the scalable coding (SVC/H.264). Rate and trafficadaptation along with admission control and automatic layer managementto optimize QoS in terms of the number of admitted sessions and videoquality in wireless video transmission is also included. Trafficmanagement, path selection, and fame filtering are included as crosslayer optimization techniques for video streaming over UDP/RTP incellular networks. The cross layer framework includes video codecoptimization, time slicing optimization for layer coded transmissions,and an adaptive Modulation and Coding Scheme (MCS) to optimize users'QoS levels and energy efficiency of wireless multimedia broadcastreceivers considering display size and energy constraints.

By its nature, compression removes redundancy from a source and is thusinherently more sensitive to transmission errors. There are severalmethods that a compression system can use to mitigate the effect oftransmission errors in a video stream. If the video stream has quality,spatial, or temporal scalability, then we can use the concept of unequalerror protection (UEP) to provide higher levels of protection for themore important information bits. On the receiver side, if the videostream has the property of error resilience, then error propagation isminimized and a good proportion of errors can be ignored. Afterdecoding, error concealment may be applied to the decoded output. Onetechnique which uses temporal error concealment is to save the previousframe and replay the frame when the current frame is corrupted. Anotherpossibility is to use the surrounding pixels of an area which is inerror to predict the current pixels. Typically, error concealment isused as a last resort if the other methods fail.

FIG. 1 is a block diagram illustrating a wireless data transmissionsystem 100, according to some example embodiments of the presentinvention. The wireless data transmission system 100 includes atransmitter 102, configured to receive, at an input terminal 103, data(e.g., uncompressed video data) from a data source 104 (e.g., anexternal data source such as a video card computer system). The wirelessdata transmission system 100 further includes a receiver 106. Thereceiver 106 may be incorporated into a user device or electronic device108 (e.g., an electronic display device, smart phone, television,tablet, etc.) having a display panel 110 configured to display imagesand video. The display panel 110 includes a plurality of pixels, each ofwhich may include a plurality of different color components (orsub-pixels) (e.g., each pixel may include a red component, a greencomponent, and a blue component).

In the context of video, each frame of video is displayed (e.g.,displayed briefly) as an image on the display panel 110. In someembodiments, the display panel 110 may, for example, be included as partof any electronic device 108 with an integrated display, such as atelevision, a monitor, a cellular phone, a tablet computer, a wearabledevice, augmented reality (AR) headset, or a virtual reality (VR)headset.

Additionally, the transmitter 102 is in wireless data communication withthe receiver 106 by way of wireless radios 112 and 114 incorporated intothe transmitter 102 and receiver 106, respectively. Thus, thetransmitter 102 and the receiver 106 are configured to transmit databack and forth between each other using any suitable wireless dataspectrum and standard (e.g., the Institute of Electrical and ElectronicsEngineers (IEEE) 802.11 standard). For example, the transmitter 102 isconfigured to transmit the data signal, along with parameters orcharacteristics of the cross-layer image optimization scheme, to thereceiver 106 by way of a direct (e.g., data) channel 130 (e.g., awireless communication channel), and the receiver 106 is configured totransmit feedback data or a return signal (e.g., including channel orvisual quality data) to the transmitter 102 by way of a return channel132.

Among other elements, the transmitter 102 further includes anapplication (APP) layer 116 (including, for example, display protocoland video encoder modules), a media access control (MAC) layer 118, anda physical (PHY) layer 120. According to embodiments of the presentinvention, various aspects of the cross-layer image optimization areperformed or executed in the APP layer 116, the MAC layer 118 and thephysical layer 120. Similarly, the receiver 106 includes an APP layer122, a MAC layer 124, and a PHY layer 126.

The wireless data transmission system 100 may further include across-layer image optimization (CLIO) module or engine 134 operating aspart of, or in coordination with the transmitter 102, for controllingthe system and method for cross-layer image optimization. The CLIOmodule 134 may include, for example, a processor, and a memory coupledto the processor, with the memory storing instructions that, whenexecuted by the processor cause the processor to execute variousoperations of the cross-layer image optimization system and methoddescribed herein. For example, the CLIO module 134 may be in electroniccommunication with the APP layer 116, the MAC layer 118, and the PHYlayer 120 to exchange data and commands back and forth for implementingthe cross-layer image optimization system and method described herein.

Thus, as illustrated in FIG. 1, the wireless data transmission system100 is configured to receive data at a transmitter 102 and transmit datato the receiver 106, after implementing various mechanisms forcross-layer image optimization in the APP layer 116, the MAC layer 118,and the PHY layer 126.

A video compression/decompression system, or codec, can be evaluated inessentially six different ways: compression, quality (distortion),complexity, latency, quality scalability, and error resilience.

According to some example embodiments of the present invention, alayer-based compression architecture may be utilized for data or videocompression, in which a video or data stream is parsed or divided intomultiple quality layers, resolution layers, spatial positions, or colorcomponents. In related-art layer-based compression architectures (e.g.,JPEG2000), a data stream begins with a frame header followed by asequence of packet headers and data bits. However, the stream may not besorted in order of importance, especially for real-time UHD codecs.

In one embodiment, a layer-based compression system (such as, or similarto the JPEG2000 standard) is utilized as a layer-based compression aspart of the cross-layer image optimization system and method (e.g.,executed by the CLIO module 134). A layer-based encoder according toembodiments of the present invention may employ the following steps: acolor transform for the components, a wavelet transform on each colorchannel, quantization of wavelet coefficients, and finally arithmeticcoding of rectangular areas of the wavelet coefficients known ascode-blocks. Such an encoder provides flexibility for the structure ofthe encoded data, by allowing various sequences of these code-blocksbased on color components, resolution levels, spatial block position, orlayers. Individual code-blocks may be independently decodable, but theheaders for these code-blocks may be dependent on previous headers andaccordingly may be decoded in sequence,

A layer-based compression encoded data stream utilized according toembodiments of the present invention may begin with a frame headerfollowed by sequence of packet headers and data bits. As noted, the datastream can be divided in multiple quality layers, resolution levels,spatial position, and color components. However, depending on thespecific encoding system utilized (e.g., JPEG2000) the stream may not besorted in order of importance, especially for real-time UHD codecs. Theexample in FIG. 2 shows an example of a layer-based compression encodeddata stream which was created in layers; for example the first layercould correspond to a compression level of 8:1; at the end of the secondlayer (having decoded all of the first layer packets) the compression isset at 4:1. Finally, decoding the third layer packets may be equivalentto sending stream at 2:1 compression. Embodiments of the presentinvention are not limited to the compression ratios or number of layersdiscussed above. Rather, embodiments of the present invention mayutilize any suitable number of layers having any suitable compressionratio according to the design and function of the cross-layer imageoptimization system and method, such that the first or highest layer hasthe highest compression ratio, the next layer has data that, when addedto the first layer, results in a compression ratio lower than that ofthe first layer, and so on, until the lowest or last layer has datathat, when added to each of the preceding layers, results in acompression ratio that is relatively low, visually lossless, ornear-visually lossless (e.g., 1:1, 1.5:1, 2:1) to viewers.

According to some embodiments, in the context of a VR display, whenstereoscopic images are transmitted to the receiver, errors in datatransmission may impact both the left and right image, even if the erroris only on one side. Accordingly, in some embodiments of the presentinvention, the cross-layer image optimization system and method (e.g.,the CLIO module 134 and/or the video encoder module of the APP layer116) may perform compression on the left image plus the difference ofthe right image, and then ensure that the various left image plus thedifference of the right image slices are encoded as a multiplexedstream. Such a procedure may reduce (e.g., by half) the number ofinstances where an error in data transmission impacts the visibility ofthe artifacts.

Embodiments of the present invention in cross layer algorithms forimproving the quality of video streaming may further utilize one or morefeatures of the PHY layer, for example, Dynamic Adaptive Modulation andCoding (AMC), in which the modulation and coding scheme (MCS) is changedbased on an indicator of channel quality in the PHY layer. A lower MCSvalue provides a lower bit error rate but also achieves lowerthroughput. The MAC layer can inform the APP layer about the MCS toadjust compression rate based on the MCS.

Losing different bits and packets have different impact on video qualityand embodiments of the present invention operate to protect moreimportant bits in video reception utilizing UEP approaches, includingmore protection for most significant bits (MSB) inuncompressed/compressed video, more protection for packet headers inJPEG 2000, or more protection for the I-frame in MPEG-4. Protection canbe achieved by using more robust MCS, retransmission, or additionalerror correction code like Reed-Solomon.

FIG. 3 is a block diagram illustrating further details of an examplearchitecture and some components of a cross-layer optimization system100, which includes control schemes at the APP, MAC and PHY layers,according to some example embodiments of the present invention. Themeasurement and feedback block is used to estimate conditions like thebit error rate, video quality, and the available bandwidth of certainservice class. Then, the models in the APP layer and the MAC layer canminimize the distortion of the video quality by optimal bit allocation,reducing the number of bits required for forward error correction, anddetermining the priorities of packets according to the impact of theirloss.

In some embodiments, as illustrated in FIG. 3, in the APP layer, thesystem may enable/disable light compression based on the feedback andmeasurement from the receiver (e.g., based on data received on thereturn channel 132 from the receiver 106). The feedback can be based onthe video quality, for example, and the measurement of the video qualitycan be based, for example, on the signal-to-noise ratio (SNR) or peaksignal-to-noise ratio (PSNR) reported by the receiver.

The MAC layer in FIG. 3 may include many functions, modules, orprocedures including, for example, pixel partitioning, retransmissionand multiple CRC checks, packetization with features like MSB/LSBseparation and different buffers. Based on feedback received from thePHY layer or parameters in the APP layer, the system (e.g., the CLIOengine 134) can change parameters of these functions in the MAC layerand decide how these functions, modules, or procedures should work. Forexample, if pixel partitioning is performed or not and if pixelpartitioning is performed, how the partitions are divided into packets.Also, they system may be configured to decide or select whether or notretransmission is enabled.

According to some embodiments, the system (e.g., the CLIO engine 134)may utilize these features in UEP implementation. For example, some ofthe packets may have these features like retransmission or thesefeatures can have different parameters for different packets likedifferent MCS or MSB/LSB separation.

The PHY layer (e.g., PHY layer 302, with a corresponding PHY layer 304in the receiver) can incorporate multiple layers for implementation ofUEP as well, as illustrated in FIG. 3. Different packets may be sentthrough different layers with different MCS values. Different layers canbe associated with different buffers with different priorities as well.According to some example embodiments, the system (e.g., the CLIO engine134) may associate different packets according to the video quality andmeasurement feedback to different layers and buffers.

The first step in cross layer image optimization for uncompressed orcompressed transmission, according to some example embodiments of thepresent invention, is layering or partitioning. Embodiments of thepresent invention utilize a plurality of predefined layering and/orpartitioning profiles, described in further detail below. As part of adata transmission session, one of the different layering and/orpartitioning profiles is selected based on one or more predefinedparameters including, for example, the type of display device, differentcriteria or indicators for video visual quality (including peaksignal-to-noise ratio (PSNR)), channel quality, codec requirements orselection, MCS and data rate requirements, and the like. Each profileindicates or is defined by a different set of procedures in thetransmitter and receiver including, for example, a different errorcorrection technique, compression, or packetization.

Additionally, embodiments of the present invention may utilize anunequal error protection (UEP) method as part of the cross-layer imageoptimization for ordered bit streams of different layers. Ordering ofbit streams or packets may be performed on layered-based compressedvideo or uncompressed video. Different MCS values are utilized fordifferent layers according to channel quality.

According to some example embodiments of the present invention, dynamictagging for packets (e.g., in IEEE 802.11ad) may be utilized as part ofthe cross-layer image optimization system and method of the presentinvention. Importance levels and corresponding information may betransmitted or signaled to the receiver, as importance level awareprocedures are implemented in MAC packetizer for the transmitter anddepacketizer at the receiver.

As described above, embodiments of the present invention, the system mayutilize dynamic layering and partitioning profiles as part of thecross-layer image optimization system and method of the presentinvention. The transmitter and receiver (e.g., the transmitter 102 andthe receiver 108) may both utilize or agree on a plurality of predefinedlayering/partitioning profiles. These profiles can be selected based ondifferent criteria for video visual quality (including PSNR), channelquality, codec requirements or selection, MCS, data rate requirements,etc. The receiver then selects the corresponding algorithm(s) to be runto reverse the layering method and/or further correct any transmissionerrors using an error concealment algorithm corresponding to theselected profile.

In one embodiment, the system utilizes 7 profiles for use with variouschannel conditions, although the number of profiles is not limitedthereto, and any suitable number of profiles may be utilized accordingto the design and function of the wireless data transmission system.

For example, a profile “00” may be defined for when the channel isoperating in ideal or near ideal conditions, while a profile “06” may bedefined for when the channel is operating in very poor conditions. Theprofiles between “00” and “06” are then defined for various intermediaryconditions.

For example, one profile (e.g., profile “00” or the ideal/near-idealcondition profile) can be defined for uncompressed video transmissionfor when the channel quality is high. For such a profile, the pixelpartitioning technique may assume that pixels beside each other have ahigh probability of containing very similar content. If any pixel from agroup of 4 is missed, it might be recalculated from other 3 capturedpixels. Profile “00” can be selected by dividing an image to 4 types ofpixels, as illustrated in FIG. 5. Three colors of pixels in one type arepacked in 1 packet. All the four type packets are transmitted to thereceiver. The receiver, knowing the Profile “00” is being transmitted,can improve the image quality. For example, if pixel data of any of thepixels (or any color in the pixel) is missing, the receiver runs anaveraging method to replace the missing pixel or color by averaging agroup of 4 or 8 surrounding pixels (number of averaging pixel can bedynamic). Image compression methods also rely on the idea that pixelsbeside each will have correlated values. This method may be simpler toimplement and more robust to bit errors compared to an entropy-codedcompression algorithm.

A second profile (e.g., profile “01”) can be selected by dividing animage to 4 types of pixels, similar to the first profile (Profile “00”).However, three types of pixels are chosen for transmission. The systemcalculates if the resulted distortion by dropping one type of pixel islower than a threshold value. If this profile is signaled to thereceiver, the receiver will estimate the missing pixel data values fromaverage of three (or more) other pixels within each of the four pixeltypes. The thresholds can be selected based on different criteriaincluding, for example, visual quality (e.g., PSNR). The choice of pixeltype which was not transmitted and threshold can also be signaled.

A third profile (e.g., Profile “02”) may operate on (e.g., be selecteddue to) a lower bandwidth than the first two profiles described above(e.g., Profiles “00” and “01”). For example, the third profile may nottransmit the least significant bit (LSB) of pixels in one or all of thepackets. Then, in the receiver, an averaging algorithm may compensatefor those bits. Another threshold level for distortion can be consideredto select the third profile (e.g., Profile “02”) instead of the first orsecond profiles described above (e.g., Profile “01” or “00”) foruncompressed video.

Other profiles can be considered for compressed video when the qualityof channel is lower and PHY layer signals a lower value for MCS. Forexample, multiple quality-layering may be used in order to define newprofiles for wireless video transmission. In a layered-basedcompression, multiple layers can be generated, as described above.According to embodiments of the present invention, the highest layer mayinclude the data with the highest compression ratio from among theplurality of layers, and the lowest layer may include data which, ifadded to the higher layers, would form an image with a relatively low(e.g., 1:1 or 1:1.5, or any suitable or visually lossless compressionratio) compression ratio. For example, in a three layer compressionscheme (e.g., as illustrated in FIG. 2), Layer 1 may include thecompressed image data with a compression ratio of 4:1. Layer 2 mayinclude the data which if added to Layer 1 would form the compressedimage with a compression ratio of 2:1. Layer 3 may include the bitstream that if added to layer 1 and 2, the resulting image has thecompression ratio of 1:1 (or 1.5:1 or any other visually losslesscompression ratio).

The transmitter selects which layers to send according to channelquality, visual quality, codec requirements, etc. We can define a fourthprofile (e.g., Profile “03”) as sending Layers 1, 2, and 3 (or all ofthe layers), a fifth profile (e.g., Profile “04”) as sending Layers 1and 2 (or a sub-set of the layers), and a sixth profile (e.g., Profile“05”) as sending only Layer 1 or the layer with the highest compressionratio. For a seventh profile (e.g., profile “07”) the transmitter maysend a portion of one of the layers, for example, all of Layers 1 and 2and only a portion of Layer 3. The receiver, by knowing the seventhprofile (e.g., Profile “06”) is transmitted, will look for missing datainformation or run additional correcting algorithms. The transmitter canindicate the length of original packet and also transmitted packetlength in the packet header. The receiver therefore knows that part ofthe layer is not received and can infer the correct size of receivedpacket. Also, the receiver warns the decoder that part of a packet isnot received and it may request that extra information be resent orretransmitted by the transmitter.

Additional profiles may be established according to the type of databeing transmitted, the nature of the display on which images will bedisplayed, or the importance of particular portions of an image relativeto other portions of the same image. For example, according to someembodiments, certain portions or regions of the image may be moreimportant than other portions of the image in the sense that errors indata transmission for bits corresponding to one portion of the image maybe more perceptible to the viewer than errors in data transmission forbits corresponding to another portion of the image.

According to some embodiments, the cross-layer image optimization systemand method may be deployed in the context of a VR display system, forexample, in which the display (e.g., the display 110) is a VR display.In such situations, a vast majority (e.g., 90 percent) of the viewer'sperception of the image is focused on and around the center of the image(e.g., the center 15 degrees of view), because users tend to move theirhead to focus on different portions of an image. Thus, as illustrated inFIG. 5, the cross-layer image optimization system and method (e.g., theCLIO module 134) may prioritize the data corresponding to a first orcentral region of an image (e.g., the region labeled in FIG. 5 with thelabel “A,” for example, 15 degrees of view) above a second region of theimage (e.g., the region labeled in FIG. 5 with the label “B,” forexample, between the first region and a second field of view outside thefirst region, or between 15 and 20 degrees of view). Further, the firstand second regions of the image may both be prioritized above a thirdregion of the image (e.g., the region labeled “C” in FIG. 5, forexample, outside the first and second regions or outside 20 degrees ofview).

Another method of identifying regions of an image having higherimportance or priority is described in U.S. patent application Ser. No.15/288,977, which is incorporated herein by reference in its entirety.For example, according to some embodiments, data corresponding toregions of an image having higher importance may be designated with ahigher level of importance or higher degree of error protection comparedto data corresponding to regions of an image having lower importance.The regions of an image that have higher importance may vary accordingto the type of display device corresponding to the receiver, forexample, as discussed herein with respect to FIG. 5.

Because the different regions of the image may be prioritizeddifferently, the cross-layer image optimization system and method (e.g.,the CLIO module 134) may select a profile (e.g., a layering/partitioningprofile, as discussed above) or a UEP scheme (described in more detailbelow), such that the data in the first region has the lowest level oferrors compared to data in the other regions of the image, the secondregion of the image has the second lowest level of errors, and so on.Furthermore, according to some embodiments, the cross-layer imageoptimization system and method (e.g., the CLIO module 134) may partitionthe data such that errors from the outer region or regions do notpropagate to the highest priority (e.g., the center) regions of theimage.

According to some example embodiments, the cross-layer imageoptimization system may utilize, for example, the various profilesdiscussed above (for example, the first through seventh profiles) forprioritizing different portions of the image for transmission, such thata lower number profile (e.g., a lower error-prone profile) is selectedfor the higher priority portions of the image, and a higher numberprofile (e.g., a higher error-prone profile) is selected for the lowerpriority portions of the image. According to some embodiments, aseparate profile may be predefined specifically according to the type ofdisplay device (e.g., television, VR display, AR display, wirelessmedical display, and the like), such that data is transmitted withdifferent procedures in transmission (e.g., compression, encoding,packetization, bit dropping, or pruning) or reception (e.g.,decompressoin, decoding, error correction, de-packetization,retransmission) according to the priority level of the portion of theimage data being exchanged.

In one embodiment, the selection criteria for layering or partitioningprofiles can be based on video visual quality (e.g. PSNR) and/or channelquality (e.g. SNR). Layering identifies if uncompressed video,uncompressed video with some packets being dropped, or compressed videowith different compression ratios should be sent. Different layers maybe treated differently in terms of queue priority, delay tolerance,error protection, and/or reliability. Correspondingly, the transmitterand receiver, by selecting a profile or receiving a profile number, mayenable different procedures prior to or as part of data transmission(e.g., compression, encoding, packtization, bit dropping or pruning) orreception (e.g., decompression, decoding, error correction,de-packetization).

According to some embodiments, the choice of the profile can betransmitted to the receiver by a bit selection in a MAC or PHY headerpacket. Based on the bit selection, the receiver may identify theselected method for layering. In addition to the profile number, anyother information required for the selected profile might be transmittedto the receiver including, for example, the number of layers incompression, and/or number of packets/bytes in each layer. The extrainformation can be sent by bit patterns in the packet header or througha secure channel or can be pre-defined.

Thus, one or more example embodiments of the present invention include asystem and method for selecting a set of predefined layering and/orpartitioning profiles for transmitting data (e.g., uncompressed videodata) from a transmitter to a receiver over a wireless datacommunication channel. Profiles may be selected according to variousconditions or parameters including, for example, channel quality, videoquality, codec requirements or selection, MCS and data raterequirements, the nature or characteristics of the data beingtransmitted, the nature or characteristics of a display device at thereceiver end (e.g., coupled to the receiver), and the like. Theselection of the profile may be transmitted to the receiver from thetransmitter by a bit selection in the MAC layer and/or PHY layer headerpacket, for example. Depending on the selected profile, additionalinformation may also be transmitted (e.g., in the MAC layer and/or PHYlayer header packet) from the transmitter to the receiver indicating oneor more parameters for decompression, decoding, error correction, andthe like. Based on information transmitted from the transmitter to thereceiver (e.g., the bit selection) the receiver may be enabled toidentify the method and/or protocol utilized at the transmitter for datalayering and/or partitioning and/or compression, and thereby select acorresponding algorithm or procedure to execute to reverse the layeringand/or partitioning and/or compression, and also to correct any errorsduring transmission with error concealment algorithms or procedurescorresponding to the selected profile.

Embodiments of the present invention may further utilize priority-basedunequal error protection (UEP) to protect (e.g., facilitate propertransmission of) bits that are more important, for example, for thepurposes of video decoding. For example, in layer-based compression,bits associated with or corresponding to a higher compression rate(e.g., corresponding to the base layer, or Layer 1) are more importantin video decoding and therefore require more protection. In addition,bit errors in the packet headers or frame headers render the videodecoder unable to decode subsequent data bits. In one embodiment of thecross-layer image optimization system method, packet headers are movedto the beginning of each frame, as illustrated in FIG. 6. Additionally,packet headers may be identified by unique begin and end values, asillustrated in FIG. 7. When headers are moved to the beginning of theframe, the identifiers are also moved to separate the headers from eachother. The end identifier may also be removed, according to some exampleembodiments. The last end identifier can be kept to distinguish betweenend of header information and data information in the next layer.

According to some example embodiments, the cross-layer imageoptimization system and method (e.g., the CLIO module 134) may utilizedifferent protection techniques against loss or error in the wirelessmedium for those headers and the base layers. For example, according tosome embodiments, different MCS values may be utilized for differentlayers or different forward error correction (FEC) algorithms may beutilized with different coding rates. Additionally, the unequal errorprotection operations may be executed at the MAC layer, thereby enablingembodiments of the present invention to improve or optimize thereorganization engine and apply it to different codecs as well. Forexample, while the APP layer delivers the whole data to the MAC layer,the MAC layer has more flexibility to perform various cross-layer imageoptimization operations described herein. Additionally, by executing theUEP in the MAC layer, the system is relatively more dynamic, becausedifferent functions can be modified with different parameters accordingto the particular situation, as described herein. Further, embodimentsof the present invention may therefore be enabled to perform onlypartial reorganization based on the status of the wireless channel.

Compression techniques utilized according to embodiments of the presentinvention may provide additional partitioning techniques aside fromlayers of different compression ratios by providing different datacomponents as illustrated in FIG. 8. For example, various similar pixelbits (e.g., pixel bits with the same or similar MCS values) may bereorganized in a packet to be grouped together. Any suitable compressiontechnique which can also partition its output into multiple layers withdifferent priority can be used. The same approach in moving headers anddata can be used. Different MCS values or FEC with different codingrates can be used for header or partitions of data.

In one embodiment, the header includes the information on the length ofthe data packet coming after that header in a frame. In some instances,the cross-layer image optimization system and method may transmit aportion of the data bits in each layer or partition according to channelcondition, MCS conditions, bandwidth conditions, etc. Therefore, thelength of data stream may also be transmitted may be appended to theheader information and transmitted to the receiver. For example, a fixedor predetermined number of bytes (N bytes) may be reserved to identifythis length. According to some example embodiments, the length of thedata stream can be at the end of each header for easy identification.

Thus, a system and method according to some example embodiments of thepresent invention may be configured to repartition or reorganize bits ina layer or level based compression system according to priority, delaytolerance, and/or protection or reliability levels for transmission overthe wireless data communication channel. A frame of data includes aframe header and packet header bits, which require a higher level ofprotection compared to the data bits. Data bits in the frame may also bepartitioned into different priority, protection, and/or reliabilitylevels. Headers may be identified according to a predefined or known bitstring, and data bits may be sorted according to an order of importance.According to some embodiments, higher priority or less delay tolerantbits may be moved or reorganized to the beginning of a frame of data.For example, a frame of data may begin with the frame header, followedby packet headers, followed by data bits (e.g., in order of priority orimportance). Packet headers maybe separated or identified by predefinedor predetermined bit strings. Additionally, according to someembodiments, the length of data bit streams may be indicated byinserting an indicator of the length corresponding to each packet headerinto, before, or after the packet header. A decoder at the receiverreconstructs the frame, for example, by moving packet headers and otherdata bits to their original locations (prior to reorganization), andremoving any additionally inserted data, such as the data length bits,but uses the additional data to reconstruct the frame.

Accordingly, embodiments of the present invention may provideimplementation of unequal error protection for any layer and/or levelbased compression technique for cross layer image optimization. Headers,layers, and/or data partitions may be reorganized according toimportance or priority level, and such reorganization may be executed inthe MAC layer of the transmitter. According to the cross-layer imageoptimization scheme, some portion of the data from each packet may beremoved or dropped, and information about this data may be incorporatedinto the header information. Additionally, embodiments of the presentinvention may utilize unequal error protection, for example, byutilizing different MCS values for forward error correction to protectinformation with different levels of importance or differently. As partof the unequal error protection according to some example embodiments ofthe present invention, because all headers may be reorganized to begrouped together (e.g., at one location) all of the headers may undergoone unequal error protection algorithm, rather than, for example,utilizing multiple small forward error corrections for each of theheaders in a frame individually.

According to some example embodiments of the present invention, becausedifferent packets may be protected differently using UEP andlayer/partition-based compression, losing different packets affect thevisual quality differently. If certain packets in the base layer orheaders are lost, for example, the system may not be able to reconstructthe video at the receiver side. On the other hand, some packet loss mayresult in PSNR degradation. Additionally, as discussed above, dependingon the type of display device being utilized in the system, differentportions of image data may have a higher priority in terms of minimizingerrors than other portions of the image data. Therefore, embodiments ofthe present invention may further utilize dynamic UEP and importanceaware tagging, as described in more detail below.

As discussed above, the cross-layer image optimization system and methodaccording to embodiments of the present invention operate to partitionthe packets into different levels of importance based on priority, delaytolerance, protection, or reliability. Each level of importance can betreated differently in transmission and/or reception.

In one embodiment, each frame is divided into multiple video packetswith different levels of importance. The system then tags each packetwith the corresponding importance level and corresponding bits arepacked in Aggregated MAC Protocol Data Unit (A-MPDU) subframes andPhysical Layer Convergence Procedure (PLCP) Protocol Data Unit (PPDU)frames. In one example, PPDUs can be tagged with the importance leveland dynamically packed with packets and A-MPDU subframes of the sameimportance level. If retransmission is enabled, the A-MPDU subframesthat need to be retransmitted are required to be queued and placed inthe proper PPDU with proper importance level.

According to another embodiment, each A-MPDU subframe may be taggedaccording to importance level. In this case all PPDUs are consideredequally in terms of importance level and they are packed with A-MPDUswith different importance levels. In delay sensitive scenarios whichrequire retransmission, it may be advantageous to tag A-MPDU subframesrather than PPDU packets. Although tagging A-MPDU subframes rather thanPPDU packets may utilize more overhead for tag information, such anapproach may be easier for managing packets of multiple importancelevel.

According to various embodiments, a bit-pattern can be added to theheader of A-MPDU or PPDU in order to tag the packets. The bit patterncan be M bits, with 0 . . . 0 representing the highest importance and 1. . . 1 representing the lowest importance. According to someembodiments, the value M may be pre-defined or the M value may besignaled as part of the header information. According to someembodiments, the transmitter and receiver may define 2{circumflex over( )}M importance levels and different procedures for each level.

For a guaranteed quality of service for wireless video transmission overIEEE802.11ad or IEEE 802.11ay, the cross-layer image optimization systemand method according to some example embodiments of the presentinvention may implement UEP through dynamic Importance Level Aware AMCfor each tagged importance level. Different importance levels can bemodulated with different MCS values or they can be encoded withdifferent forward error correction (FEC) values or with the same FECvalues but with different coding rates.

In related art systems, the PHY may recommend the MCS index appropriatefor the current channel situation. According to some embodiments of thepresent invention, however, the cross-layer image optimization systemand method may adapt the recommended MCS for a majority of the bitstream. For example, the system may decrement the MCS index for moreimportant layers or packets, and increment the MCS index for lessimportant layers or packets. FIG. 9 illustrates an example foruncompressed video transmission in which the system selects differentMCS values for MSB/LSB parts of bits representing RGB colors in pixels.The four MSB bits are sent with MCS−1, the next 2 bits are sent withMCS, and the last 2 LSB bits are sent with MCS+1. It also includesregrouping of the bits into packets/A-MPDU subframes such that bits withthe same MCS index are grouped into the same packets/A-MPDU subframes.

For example, for compressed video transmission, after layered-basedcompression is performed and the bits are reorganized according to theimportance level of transmission, as discussed above with respect toFIG. 6 and FIG. 8, the headers and base Layer 1 are transmitted withMCS−1, Layer 2 is sent with MCS, and finally Layer 3 can be sent withMCS or MCS+1. The headers and layer 1 are therefore protected more andare transmitted almost error free. Because these information bits aresent with MCS−1, it is probable that the required bandwidth exceeds thesupported bandwidth of the channel with MCS. To compensate, portions ofLayer 3 can be dropped and/or Layer 3 bits can be modulated with MCS+1

In IEEE 802.11ad, the MCS index is only signaled for each PPDU packet of262,143 bytes. Therefore, if PPDUs are tagged with importance levels,embodiments of the present invention may utilize the IEEE 802.11adprocedure for using the recommended MCS or modified MCS index for eachpacket. According to some embodiments, the signaling in IEEE 802.11admay be modified such that MCS is signaled for each A-MPDU subframe. Forexample, the current reserved bits or unused information in the headersof each packet may be utilized. According to IEEE 802.11adspecification, the header includes the fields in FIG. 10. As an example,in peer to peer connections, embodiments of the present invention use“Address 4” for signaling MCS. Embodiments of the present invention mayadditionally add new bytes to the header to signal the MCS for eachA-MPDU.

Thus, a cross-layer image optimization system and method according tosome example embodiments of the present invention may operate topartition data (e.g., pixel data, packets, etc.) of a video frame intodifferent categories of importance based on, for example, priority,delay tolerance, and/or protection or reliability. Each video frame maybe divided into multiple levels of importance and corresponding bits maybe packed into A-MPDU subframes. Different packets (e.g., A-MPDUsubframes, PPDU subframes, etc.) may be tagged according to importancelevel, and according to some embodiments, the different packets may betransmitted to the receiver according to their order of importance.Different packets (e.g., A-MPDU subframes) may be grouped (e.g., intoPPDU packets) according to their importance level. A value or tagindicating the importance level may be added to the header of thepackets. At the receiver end, the receiver may be enabled to determinethe tag level and thereby further enable different error recover and/orconcealment procedures according to the tagging. Additionally, datahaving different levels of importance may be protected differently byselecting different modulation and coding scheme (MCS) values for eachpacket. The corresponding MCS values may be inserted into the header forthe packet in addition to the tag.

Accordingly, embodiments of the present invention may enable dynamicpartitioning and/or tagging for packets (e.g., in IEEE Multi-Gigabit802.11) and importance aware procedures in the transmitter and receiverfor wireless data transmission, including AMC for wireless cross-layerimage optimization. Embodiments of the present invention may enabletagging of each small packet within the A-MPDU (data) packets, withmultiple content driven tags. The tags may be transmitted and signaledto the receiver. Different procedures or parameters may be associatedwith the tags for incorporating various aspects of the cross-layer imageoptimization mechanisms discussed herein. Additionally, embodiments ofthe present invention may utilize unequal error protection based tags,such as AMC (MCS adaptation for tags).

According to various embodiments of the present invention, the systemmay continuously monitor the various parameters and factors forcross-layer image optimization, and adjust accordingly. For example, thesystem may monitor the return signal from the receiver and ascircumstances change, adjust the cross-layer image optimizationaccordingly, for example, by selecting different profiles, compressionor error correction techniques, and the like, as discussed above, forsubsequent data transmission.

FIG. 11 is an example flow diagram illustrating various operations in amethod for cross-layer image optimization, according to some exampleembodiments of the present invention. The number and order of theoperations illustrated in FIG. 11, however, is not limited to the flowdiagram illustrated. For example, according to some embodiments, theorder of the operations may be modified (unless otherwise indicated) orthe process may include additional operations or fewer operations allwithout departing from the spirit and scope of the present invention.For example, the various other features and operations described in thepresent disclosure and the corresponding figures may be incorporatedinto the process, or certain operations may be omitted according to someexample embodiments of the present invention.

The process starts and, at 1200, the system (e.g., the wireless datatransmission system 100 and/or the transmitter 102) receives data (e.g.,a video data signal including, for example, uncompressed video data)from a data source. At 1202, return data is received from a receiver(e.g., receiver 106), which includes various information about the datatransmission to the receiver including, for example, channel quality,visual quality, and the type of display device connected to thereceiver. Thus, at 1204, the system may identify the type of displaydevice. At 1206, a layering/partitioning profile is selected based onone or more predefined parameters including, for example, the type ofdisplay device, different criteria or indicators for video visualquality (including peak signal-to-noise ratio (PSNR)), channel quality,codec requirements or selection, MCS and data rate requirements, and thelike.

At 1208, layer or level-based compression may be performed on the data,for example, according to the selected layering/partitioning profile.Then, at 1210, the layers, partitions, packets, or bits of data arereorganized into groups or packets according to their importance level,as discussed above.

At 1212, the data may be tagged to indicate the importance level of thedata and/or the packets of data. Then, based on the importance level ofdifferent types of data, unequal error protection may be performed fortransmitting the data to the receiver. At 1216, the data is transmittedto the receiver, for example, for display on a display panel.

A wireless data transmission system and/or any other relevant devices orcomponents according to embodiments of the present invention describedherein may be implemented utilizing any suitable hardware, firmware(e.g. an application-specific integrated circuit), software, or asuitable combination of software, firmware, and hardware. For example,the various components of the wireless data transmission system may beformed on one integrated circuit (IC) chip or on separate IC chips.Further, the various components of the display device may be implementedon a flexible printed circuit film, a tape carrier package (TCP), aprinted circuit board (PCB), or formed on a same substrate as thedisplay device. Further, the various components of the wireless datatransmission system may be a process or thread, running on one or moreprocessors, in one or more computing devices, executing computer programinstructions and interacting with other system components for performingthe various functionalities described herein. The computer programinstructions are stored in a memory which may be implemented in acomputing device using a standard memory device, such as, for example, arandom access memory (RAM). The computer program instructions may alsobe stored in other non-transitory computer readable media such as, forexample, a CD-ROM, flash drive, or the like. Also, a person of skill inthe art should recognize that the functionality of various computingdevices may be combined or integrated into a single computing device, orthe functionality of a particular computing device may be distributedacross one or more other computing devices without departing from thescope of the exemplary embodiments of the present invention.

Although this invention has been described in certain specificembodiments, those skilled in the art will have no difficulty devisingvariations to the described embodiment, which in no way depart from thescope and spirit of the present invention. Furthermore, to those skilledin the various arts, the invention itself herein will suggest solutionsto other tasks and adaptations for other applications. It is theApplicant's intention to cover by claims all such uses of the inventionand those changes and modifications which could be made to theembodiments of the invention herein chosen for the purpose of disclosurewithout departing from the spirit and scope of the invention. Thus, thepresent embodiments of the invention should be considered in allrespects as illustrative and not restrictive, the scope of the inventionto be indicated by the appended claims and their equivalents rather thanthe foregoing description.

What is claimed is:
 1. A method for transmitting video to a displaydevice comprising: transmitting, by a transmitter, a first data signalover a wireless communication channel to a receiver of the displaydevice; receiving, by the transmitter, a return signal transmitted bythe receiver, wherein the return signal is based on the first datasignal; selecting, by the transmitter, at least one profile for sendingthe data signal according to the return signal, wherein the at least oneprofile is selected from a plurality of profiles based on the returnsignal, wherein the plurality of profiles comprise: a plurality of pixelpartitioning profiles configured to divide an image into types ofpixels; and a plurality of layering profiles configured to divide theimage into layers, wherein a higher layer comprises data with a highcompression ratio, and a lower layer comprises data that when added tothe data with the high compression ratio, forms an image with acompression ratio that is lower than the high compression ratio; andtransmitting, by the transmitter, a second data signal to the receiveraccording to the at least one profile for display on at least onedisplay panel of the display device.
 2. The method of claim 1, furthercomprising: identifying, based on the return signal, a display devicetype; and selecting, by the transmitter, the at least one profileaccording to the display device.
 3. The method of claim 2, wherein thedisplay device type comprises at least one of a virtual reality deviceor an augmented reality device.
 4. The method of claim 3, wherein the atleast one profile comprises a first profile for a first region of the atleast one display panel and a second profile for a second region of theat least one display panel.
 5. The method of claim 4, wherein the firstregion has a higher priority than the second region.
 6. The method ofclaim 5, wherein the first region comprises a region with a highervisual perception for a viewer of the display device than the secondregion.
 7. The method of claim 6, further comprising applying differenttransmission parameters including at least one of: a first unequal errorprotection scheme to the first region and applying a second unequalerror protection scheme to the second region; a first compression schemeto the first region and applying a second compression scheme to thesecond region; and a first modulation and coding scheme to the firstregion and applying a second modulation and coding scheme to the secondregion.
 8. The method of claim 1, wherein: the second data signalcomprises a stereoscopic data signal having a first image and a secondimage; and the display device comprises a stereoscopic display device.9. The method of claim 8, further comprising: compressing, by thetransmitter, the first image; compressing, by the transmitter, adifference of the first image and the second image; and multiplexing, bythe transmitter, the compressed first image and the compresseddifference, wherein transmitting, by the transmitter, the data signalcomprises transmitting the multiplexed compressed first image and thecompressed difference.
 10. The method of claim 1, further comprisingtransmitting, by the transmitter, profile information regarding theselected at least one profile to the receiver.
 11. A method of receivingvideo at a display device comprising: receiving, at a receiver, a firstdata signal over a wireless communication channel; generating, by thereceiver, a return signal based on the first data signal; transmitting,by the receiver, the return signal to the receiver; receiving, by thereceiver, a second data signal comprising a profile informationregarding a profile selected by the transmitter, wherein profilecomprises a pixel partitioning profile or a layering profile;performing, by the receiver, a delayering procedure when then theprofile information corresponds to the pixel partitioning profile, or adepartitioning procedure when the profile information corresponds tolayering profile; and generating an image based on the second datasignal.
 12. The method of claim 11, wherein the return signal comprisesat least one of an indicator of a quality of the wireless communicationchannel as measured by the receiver, an indicator of visual quality asmeasured by the receiver, an indicator of compression quality asmeasured by the receiver, or an indicator of environment as measured bythe receiver.
 13. The method of claim 12, further comprising:reconstructing, by a decoder at the receiver, a video frame from thedata signal, by at least one of moving header and data bits to theiroriginal locations, and removing any an additionally inserted data. 14.The method of claim 12, wherein the profile information is received bythe receiver via at least one of a bit pattern in a packet header, asecure channel, or is pre-defined.
 15. The method of claim 11, furthercomprising correcting, by the receiver, a transmission error using anerror concealment procedure corresponding to the selected profile. 16.The method of claim 11, wherein the at least one profile comprises afirst profile for a first region of the at least one display panel and asecond profile for a second region of the at least one display panel.17. The method of claim 16, wherein the first region has a higherpriority than the second region.
 18. The method of claim 17, wherein thefirst region comprises a region with a higher visual perception for aviewer of the display device than the second region.
 19. The method ofclaim 11, wherein the second data signal comprises a compressedstereoscopic data signal having a compressed first image and acompressed difference, and further comprising: demultiplexing, by thereceiver, the compressed first image and the compressed difference; anddecompressing, the compressed first image and the compressed difference.20. A system for transmitting data for a display device comprising: atransmitter configured to communicate to a receiver of the displaydevice, wherein the transmitter is configured to: transmit a first datasignal over a wireless communication channel to a receiver of thedisplay device; receive a return signal transmitted by the receiver;select at least one profile for sending the data signal according to thereturn signal, wherein the at least one profile is selected from aplurality of profiles based on the return signal, wherein the pluralityof profiles comprise; a plurality of pixel partitioning profilesconfigured to divide an image into types of pixels; and a plurality oflayering profiles configured to divide the image into layers, wherein ahigher layer comprises data with a high compression ratio, and a lowerlayer comprises data that when added to the data with the highcompression ratio, forms an image with a compression ratio that is lowerthan the high compression ratio; and transmit a second data signal tothe receiver according to the at least one profile for display on atleast one display panel of the display device; and wherein the receiveris configured to: generate the return signal based on the first datasignal; transmit the return signal to the receiver; receive a seconddata signal comprising a profile information regarding a profileselected by the transmitter, wherein profile comprises a pixelpartitioning profile or a layering profile; perform a delayeringprocedure when then the profile information corresponds to the pixelpartitioning profile, or a departitioning procedure when the profileinformation corresponds to layering profile; and generate an image basedon the second data signal.